Skip to content

feat: add postgresql support#371

Open
fabioluciano wants to merge 9 commits into
apache:mainfrom
fabioluciano:feature/postgresql-support
Open

feat: add postgresql support#371
fabioluciano wants to merge 9 commits into
apache:mainfrom
fabioluciano:feature/postgresql-support

Conversation

@fabioluciano
Copy link
Copy Markdown

@fabioluciano fabioluciano commented May 6, 2026

Add PostgreSQL Support and Production Hardening

Summary

Major enhancement adding PostgreSQL database support alongside existing MySQL, plus comprehensive production hardening features including security controls, monitoring, backup capabilities, and extensive test coverage.

Key Features

Database Support

  • PostgreSQL Support: New StatefulSet template with optimized configuration for DevLake workloads
  • MySQL Improvements: Separated StatefulSet template, enhanced configuration management
  • External Database: Improved support for external MySQL/PostgreSQL instances
  • Secret Management: Enhanced database credential handling with External Secrets Operator integration

Production Readiness

  • High Availability: HorizontalPodAutoscaler (HPA) configuration for lake/grafana pods
  • Backup & Recovery: CronJob-based backup with configurable retention and storage
  • Resource Management: CPU/memory requests and limits for all components
  • Validation: Pre-install validation job checks configuration before deployment

Security Enhancements

  • Network Policies: Granular pod-to-pod communication controls (database, lake, ui)
  • Service Account Hardening: automountServiceAccountToken: false by default
  • Secret Scanning: GitHub Actions workflow to detect leaked credentials
  • Security Scanning: Trivy-based vulnerability scanning for Helm templates and containers
  • Pod Disruption Budgets: Ensure minimum availability during voluntary disruptions

Observability

  • ServiceMonitor: Prometheus scraping configuration for lake and grafana metrics
  • Gateway API: HTTPRoute support for modern ingress patterns
  • Enhanced NOTES.txt: Improved post-install instructions with connection details

Testing & CI/CD

  • Smoke Tests: Dedicated test scripts for MySQL, PostgreSQL, and E2E scenarios
  • Matrix Testing: All database combinations tested in CI (mysql-builtin, mysql-external, postgresql-builtin, postgresql-external)
  • License Compliance: Automated Apache license header checks
  • Dependency Management: Dependabot for GitHub Actions updates
  • PR Automation: Auto-labeling and workflow improvements

Documentation

  • Comprehensive README: Migration guide, architecture details, troubleshooting
  • Chart README: Auto-generated from values.yaml with helm-docs
  • Schema Validation: values.schema.json for configuration validation
  • NOTES.txt: Context-aware post-install guidance

Breaking Changes

None - fully backward compatible with existing MySQL deployments

Migration Path

Existing MySQL users: no action required
New users: choose mysql.enabled: true or postgresql.enabled: true

Testing

  • ✅ MySQL builtin deployment
  • ✅ MySQL external deployment
  • ✅ PostgreSQL builtin deployment
  • ✅ PostgreSQL external deployment
  • ✅ Smoke tests for all database types
  • ✅ E2E API/UI endpoint verification
  • ✅ Security scanning (Trivy)
  • ✅ License compliance checks

Major additions:

  • templates/statefulset-postgresql.yaml - PostgreSQL StatefulSet
  • templates/statefulset-mysql.yaml - Separated MySQL StatefulSet
  • templates/networkpolicy-*.yaml - Network isolation policies
  • templates/hpa.yaml - Horizontal pod autoscaling
  • templates/backup-*.yaml - Backup CronJob and PVC
  • templates/pre-install-validation.yaml - Pre-flight checks
  • .github/scripts/*-smoke-test.sh - Database-specific test suites
  • values.schema.json - Configuration schema validation

fabioluciano and others added 2 commits May 6, 2026 11:12
- Introduced `httpRoute` configuration in `values.schema.json` and `values.yaml` to enable Gateway API HTTPRoute with customizable properties such as `gatewayName`, `hostnames`, and `prefix`.
- Updated `values.yaml` to enable Grafana by default and set MySQL as the database type.
- Added HTTPRoute resource for routing traffic to Grafana and UI services.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@fabioluciano fabioluciano force-pushed the feature/postgresql-support branch from 192111a to f832cb4 Compare May 6, 2026 16:17
- Remove deprecated grafana.enabled=false requirement for PostgreSQL
- Update note about Grafana compatibility with both MySQL and PostgreSQL
- Add Gateway API HTTPRoute deployment scenario documentation
- Regenerate charts/devlake/README.md with helm-docs to reflect:
  - httpRoute configuration options (9 new parameters)
  - grafana.enabled default changed to true
  - database.type default to mysql
  - database.externalPort updated to match database type
Grafana dashboards only support MySQL datasources. When using PostgreSQL,
users must disable Grafana or use external Grafana instance.
- Increase MySQL startup probe failureThreshold to 60 (310s total)
- Add comprehensive database smoke tests:
  - MySQL: 6 tests (connectivity, version, charset, DB ops, user access)
  - PostgreSQL: 7 tests (connectivity, version, encoding, table ops, permissions, config)
- Fix workflow to wait for DB pods specifically, not full stack
- Add helm install timeout (10m)
- Fix secret names in smoke tests (use ${RELEASE_NAME}-db-auth)

Fixes MySQL startup probe timeout in CI environments.
@fabioluciano
Copy link
Copy Markdown
Author

This need to be approved first

apache/incubator-devlake#8870

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant